Extracting structure from human-readable semistructured text

The explosion of Web data over the past fifteen years has fostered a rich body of research in extracting structure from semistructured HTML and XML documents. Today, we are also in the middle of an explosion of semistructured documents that originate from outside the Web domain. This latter kind of semistructured data is everywhere: in electronic medical records (EMRs), government reports, digital humanities archives, and datasets from many other domains. In contrast to HTML or XML, semistructured text that has not been marked up is typically human-readable (HR), and its structure implicitly reflects a schema. Our high-level goals are to (1) explicitly recover meaningful structure in semistructured text corpora, and (2) demonstrate that this effort enables more accurate analytics and facilitates or enriches other applications. To motivate our approach and work, we highlight the differences between HR and marked-up semistructured text and illustrate concrete use cases of HR semistructured text. We identify specific instances of implicit structure along a spectrum of “structuredness” commonly found in HR semistructured text and describe a corresponding array of methods for extracting structured features. It has not been our goal to develop “the best” such methods, but to instead present a principled framework for evaluating and combining multiple extraction methods in the context of specific data analytic tasks. We also present example applications that can be quickly built on top of a substrate of features extracted from semistructured text. For concreteness, we focus this report around semistructured text found in EMRs and evaluate our framework using a large corpus of text reports from real EMRs exhibiting highly heterogeneous schema.